In many applications, an operator of a system for surveillance and remote tracking of objects controls a remote image sensor via a communication link. Examples are traffic control, border control, search and rescue operations, land surveys, police surveillance, military applications, etc.
In general, a system for surveillance and remote tracking of objects comprises a control unit at one end and a remote sensing unit at the other end which communicate over a communication link. The sensing unit, with the help of an image sensor, can be used for surveying a scene including one or more objects, and transmitting sensing-data, which includes data that was acquired by the sensing unit or data generated by the sensing unit in relation to the acquired data (e.g. captured images, object-data characterizing identified objects in the captured images etc.), to the control unit where the images can be displayed on a display for viewing by an operator. Furthermore, the sensing unit can be configured to locate and track a sighted object. The control unit provides to the sensing unit control data, including for example, different types of commands, such as lock and/or track commands, zoom commands, centering commands, etc.
According to one possible scenario, in case an operator of the control unit decides that it is desirable to track an object in the surveyed scene, he initiates a sequence of operations directed for that purpose. The operator can first send command including pointing instructions (i.e. a type of control data) to the sensing unit which identifies the object that should be tracked. The pointing instructions can include for example, “move up”, “move right”, “zoom” or similar instructions. In response, the sensing unit acts upon these instructions, and directs the image sensor towards the required area.
The operator can send additional control data including for example a lock command instructing the sensing unit to lock on a desired object. In response, the sensing unit receives the instructions and attempts to lock onto the object indicated in the command.
Once the object has been locked, sensing unit takes over command and commences to operate in response to tracking instructions, which are generated within the sensing unit and are directed for tracking the locked object. The tracking instructions are forwarded to the image sensor which in turn tracks the moving object and keeps the object in the center of FOV of the display, even while the object moves relative to the sensing unit.
In many applications, there is a time-delay between the time when the sensing unit acquires an image of an object, to when the image is displayed on the display located at the control unit, and further to the time the corresponding instructions are received at the sensing unit. Factors that can contribute to the delay include for example, signal processing, image compression/decompression, duration of the communication, and/or link bandwidth limitations. Consequently, when taking into account the delayed reaction time of the operator, the accumulated delayed time can be from fractions of a second to several seconds.
Due to this time-delay, the location of the object as displayed on the display at the control unit is generally not the current location of the object. The location displayed on the display is the location of the object before the transfer of the sensing-data from the sensing unit to the control unit (e.g. x seconds ago). Additionally, by the time the sensing unit receives the control data from the control unit and generates the instruction for the image sensor, an additional time-delay occurs, (e.g. an additional y seconds). Consequently, by the time image sensor is instructed to locate the object, the object may no longer be in the same location it was when the image picture was taken over x+y seconds ago. This may result from the motion of a mobile object, from the motion of the image sensor device or from a combination of these two factors.
Clearly, this time-delay complicates the efforts to lock onto the object. The operator has to accurately calculate and estimate the expected location of the object at a time in the future when the instructions arrive at the sensing unit. Only then is the sensing unit directed to the calculated estimated location, and a lock and tracking operation can be initiated.
If the calculation of the estimated location is not sufficiently accurate, the sensing unit will lock onto some other background object in which case the entire estimation, calculation and lock process has to be repeated. As such, the effect is a continuous feedback control loop with delay, a situation which is liable to suffer from overshoots and instability.
When a human operator inputs the lock and/or track commands the process becomes further complicated. Humans do not function well in feedback loops with time-delay and their reactions and directions are less precise than, for example, computer or processor generated instructions.
Publications considered to be relevant as background to the presently disclosed subject matter are listed below. Acknowledgement of the references herein is not to be inferred as meaning that these are in any way relevant to the patentability of the presently disclosed subject matter.
U.S. Pat. No. 7,184,574 discloses a tracking apparatus including a sensor tracker and a control tracker. The sensor tracker is connected to a sensor which senses a scene having at least one object therein, the sensor tracker provides sensor movement instructions to the sensor, enabling it to track a selected object. The control tracker is located remotely from and communicates with the sensor tracker. Additionally, the control tracker takes measurements regarding the selected object and provides tracking instructions to the sensor tracker. The sensor tracker then utilizes the tracking instructions to adjust the sensor movement instructions, when necessary.
US Patent Publication No. 2008267451 discloses a method for tracking an object that is embedded within images of a scene, including: in a sensor unit that includes a movable sensor, generating, storing and transmitting over a communication link a succession of images of a scene. In a remote control unit, the succession of images is received. Also disclosed is receiving a user command for selecting an object of interest in a given image of the received succession of images and determining object-data associated with the object and transmitting through the link to the sensor unit the object-data. In the sensor unit, the given image of the stored succession of images and the object of interest using the object-data are identified, and the object in the other image of the stored succession of images is tracked. The other image is later than the given image. In case the object cannot be located in the latest image of the stored succession of images, information of images in which the object was located are used to predict estimated real-time location of the object, and direction command is generated to the movable sensor for generating real-time image of the scene and locking on the object.
EP Patent No. 0423984 discloses a synergistic tracker system which includes both a correlation tracker and an object tracker for processing sensing-data input and for generating tracking error signals. The operation of the synergistic tracker system is controlled by a central processing unit. The system operates by first correlating a reference region image with a portion of a current digitized image provided by analog to digital converter. Secondly, the object tracker provides a precisely defined track point for an object within the current image. The correlation tracker stabilizes and limits the portion of the digitized image that the object tracker must operate upon. Stabilizing and limiting this portion of the digitized image reduces the object tracker's sensitivity to background clutter and sensitivity to a loss of lock induced by sensor motion. The object tracker provides a non-recursive update for the correlation's reference region image. The correlation tracker and the object tracker are used simultaneously and cooperatively so that the strengths of one tracker are used to overcome the weaknesses of the other. This invention provides a greater tracking tenacity, a reduction in tracker angle noise, and a reduction in hardware complexity.
U.S. Pat. No. 7,620,483 relates to a method for guiding from a remote control unit a vehicle towards a target object, said remote control unit communicating with the vehicle by means of a lagged communication channel, comprising: At the vehicle: (a) Periodically capturing frame images by a camera, assigning to each of said captured frames an associated unique time stamp, and saving within a storage at the vehicle full frame data or partial frame data of captured frames and their associated time stamps; (b) For a plurality of saved frames, sending to the control unit via the lagged communication channel full frame data, partial frame data or a combination thereof with the corresponding associated time stamp for each sent frame so that approximate or exact version of the sent frames can be reconstructed and displayed at the control unit; At the control unit: (c) Receiving said frame data and associated time stamps, sequentially reconstructing frame images from each said sent full and/or partial frame data, and displaying the reconstructed images on a display; (d) Upon marking by an operator at the control unit a point on a specific displayed frame, sending coordinates indication relating to said marked point as appearing on said specific frame or on a reference frame available at the control unit, and the time stamp associated with said specific or reference frame, as is the case, to the vehicle; At the vehicle: (e) Receiving said coordinates indication as marked and the sent frame time stamp; (f) Given the coordinates indication and frame time stamp as received, fast forward tracing said point or object coordinates from the said frame towards the most recently available captured frame, thereby finding the coordinates of the same point or object as appearing in the most recently available captured frame; and (g) Providing the coordinates of the target point or object within the most recently available captured frame, as found, to an inner guidance sub-system of the vehicle, for enabling it to track said object.
General Description
According to one aspect of the presently disclosed subject matter there is provided a tracking system comprising:
a sensing unit comprising an image sensor, the sensing unit configured to communicate over a communication link with a control unit, configured to capture two or more images of a scene and to generate a succession of images; wherein
the sensing unit is configured to identify one or more moving objects in the succession of images; to assign an object-tag to at least one of the moving objects to yield in a respective tagged object to be sent to the control unit;
wherein the sensing unit is further configured, responsive to a command received from the control unit indicative of at least one selected object, to identify a location of the selected object in a latest available captured image, the identification comprising:
determining whether the at least one selected object is a moving object or a stationary object;
in case the at least one selected object is a moving object, obtaining from the received command an object tag corresponding to the at least one selected object, and identifying the at least one selected object in the latest available captured image;
in case the at least one selected object is a stationary object, obtaining from the received command an image or piece thereof comprising the at least one selected object and identifying the at least one selected object in the latest available captured image.
According to another aspect of the presently disclosed subject matter there is provided a method of tracking objects, the method comprising: in a sensing unit comprising an image sensor: capturing two or more images of a scene and generating a succession of images; identifying one or more moving objects in the succession of images; assigning an object-tag to at least one of the moving objects to yield in a respective tagged object to be sent to a control unit communicating with the sensing unit; identifying, responsive to a command received from the control unit indicative of at least one selected object, a location of the selected object in a latest available captured image, the identification comprising: determining whether the at least one selected object is a moving object or a stationary object; in case the at least one selected object is a moving object, obtaining from the received command an object tag corresponding to the at least one selected object, and identifying the at least one selected object in the latest available captured image; in case the at least one selected object is a stationary object, obtaining from the received command an image or piece thereof comprising the at least one selected object and identifying the at least one selected object in the latest available captured image.
The system and method disclosed herein can further include one or more of the following features:
a) wherein the object-tag is associated with a respective moving object to yield the respective tagged object, based on information indicative of a location of the moving object in the succession of images; the method and system further enable: tracing the tagged object, from an earlier image in the succession of images to a later image in the succession of images, thereby maintaining a given object-tag associated with a respective moving object along the succession of images.
b) The method and system further enable: continuously sending a flow of images from the succession of images to the control unit.
c) The method and system further enable: in case the at least one selected object is a stationary object, matching the image or piece thereof to the latest available captured image;
identifying the at least one selected object in the image or piece thereof, in order to identifying the at least one selected object in the latest available captured image.
d) The method and system s a real-time location of the selected object; and
generating direction commands to the image sensor, such that the selected object is located in a real-time image of the scene generated by the image sensor.
e) The method and system further enable: storing information with respect to the one or more objects and respective object tags in a data-repository associated with the sensing unit.
f) The method and system further enable: in the control unit:
receiving a captured image from the sensing unit;
displaying the image on a display operatively connected to the control unit;
responsive to selection of an object in the image, determining whether the selected object is a moving object or stationary object;
in case the selected object is a moving object, identifying an object-tag assigned to the selected object and generating a command comprising the object-tag;
in case the selected object is a stationary object, generating a command comprising the image or part thereof; and sending the command to the sensing unit.
g) The method and system further enable: executing the command with respect to the selected object notwithstanding a time-delay between a time when the sensing unit acquires the image with the selected object, to a time when a corresponding command is received at the sensing unit with respect to the selected object.
According to another aspect of the presently disclosed subject matter there is provided a non-transitory computer storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform the method of tracking objects as disclosed above in the previous aspect.